The credibility of scientific research is in doubt, among lay consumer (Hornsey & Fielding, 2017) and scientist (Pashler & Wagenmakers, 2021) alike. Several tools have been proposed to combat this “crisis of confidence” (Ibid., p. 528). One such tool is the crowd science approach: “the organization of scientific research in open and collaborative projects” (Franzoni & Sauermann, 2014, p. 1). We focus on crowdsourced data analysis, also known as the many analysts or multi-analyst approach: giving the same dataset to different teams of scientists, who independently analyze it to answer the same research question and/or estimate a parameter of interest.
\(~~~~~\) According to science reformers, crowd-scientific findings that tell a consistent story should garner more confidence in the conclusions and increase public faith in science (Silberzahn et al., 2018; Uhlmann et al., 2019). Here, we ask if we can find empirical evidence for these claims: Does crowdsourcing data analysis improve lay perceptions of scientific findings?
Objectives
We explore the effects of scientific findings emerging from a crowd of researchers (vs. a typical research collaboration) on lay consumers’ posterior beliefs, perceptions of credibility, confidence in an aggregate effect size estimate, and ratings of researcher bias, error, and discretion.
\(~~~~~\) We compare the effects of providing lay consumers with a single, aggregate parameter estimate (the single-analyst condition) vs. multiple parameter estimates that (a) vary slightly and are all positive, leading to the same qualitative conclusion (the multi-consistent condition) or (b) vary widely and are of both signs, leading to differing qualitative conclusions (the multi-inconsistent condition). In all three conditions, the given estimates average to 5%.
Preregistered Hypotheses
Table 1: Predicted direction of effects
| Measure | Multi-consistent | Multi-inconsistent |
|---|---|---|
| 1. Posterior beliefs | \(~~~~~~~~~~\) | \(~~~~~~~~~~\) |
| 2. Credibility of the results | \(~~~~~~~~~~\) | \(~~~~~~~~~~\) |
| 3. Confidence in the precise estimate | \(~~~~~~~~~~\) | \(~~~~~~~~~~\) |
| 4. Researcher bias | \(~~~~~~~~~~\) | \(~~~~~~~~~~\) |
| 5. Researcher error | \(~~~~~~~~~~\) | \(~~~~~~~~~~\) |
| 6. Researcher discretion | \(~~~\) No prediction | \(~~~\) No prediction |
Note. Table 1 indicates the predicted direction of effects for all outcomes, compared to the single-analyst condition and controlling for prior beliefs (a green plus/red minus indicates a positive/negative prediction, respectively). For example, we hypothesized that, compared to a single-analyst study and controlling for prior beliefs, ratings of credibility would be greater in the multi-consistent condition and lower in the multi-inconsistent condition.
We run an experiment (N = 1,498) with three conditions
\(~\) Single-analyst
A single, aggregate parameter estimate
\(~\) Multi-consistent
Multiple parameter estimates with low variance and high consensus
\(~\) Multi-inconsistent
Multiple parameter estimates with high variance and low consensus
Experimental Design
Figure 1: Estimates relative to the single-analyst condition
In line with our hypotheses, lay consumers of multi-analyst studies with inconsistent results
\(~~~\) Have lower posterior beliefs
\(~~~\) Find the results less credible
\(~~~\) Have less confidence in the average effect size estimate
\(~~~\) Believe the results are more likely to stem from bias
\(~~~\) Believe the results are more likely to stem from error
Contrary to our hypotheses, lay consumers of multi-analyst studies with consistent results
\(~~~\) Have lower posterior beliefs
\(~~~\) Believe the results are more likely to stem from error
We found no significant effects on
\(~~~\) Credibility of the results
\(~~~\) Confidence in the effect size estimate
\(~~~\) Ratings of bias
Exploratory results
For the additional exploratory measure, lay consumers of multi-analyst studies (both with consistent and inconsistent results)
\(~~~\) Perceive greater researcher degrees of freedom
Figure 2: Distribution of prior and posterior beliefs by condition
Conclusion
\(~\) Crowdsourced data analysis has many worthy uses, but…
\(~~\) It’s highly resource-intensive
\(~\) Variability and lack of consensus may evoke negative responses
\(~~\) Surprisingly, even multiple, consistent estimates may backfire
Future Directions
\(~~~\) Perceptions of scientists?
\(~~\) Science communication and communicating uncertainty
Open Science Statement
The preregistration, survey materials, data, and code that support the findings of this study are openly available on GitHub and the OSF.
\(~~\) [Insert GitHub link here]
\(~~\) [Insert OSF link here]